Air-Writing Recognition - Part II: Detection and Recognition of Writing Activity in Continuous Stream of Motion Data

Air-writing refers to writing of characters or words in the free space by hand or finger movements. We address airwriting recognition problems in two companion papers. Part 2 addresses detecting and recognizing air-writing activities that are embedded in a continuous motion trajectory without delimitation. Detection of intended writing activities amongst superfluous finger movements unrelated to letters or words presents a challenge that needs to be treated separately from the traditional problem of pattern recognition. We first present a data set that contains a mixture of writing and non-writing finger motions in each recording. The LEAP from Leap Motion is used for marker-free and glovefree finger tracking. We propose a window-based approach that automatically detects and extracts the air-writing event in a continuous stream of motion data, containing stray finger movements unrelated to writing. Consecutive writing events are converted into a writing segment. The recognition performance is further evaluated based on the detected writing segment. Our main contribution is to build an air-writing system encompassing both detection and recognition stages and to give insights into how the detected writing segments affect the recognition result. With leave-one-out cross validation, the proposed system achieves an overall segment error rate (SER) of 1.15% for word-based recognition and 9.84% for letter-based recognition. Keywords—air-writing, fingerwriting, air-writing recognition, air-writing detection.